Segmentation system for segmenting an object in an image

ABSTRACT

The invention relates to a segmentation system for segmenting an object in an image. The segmentation system is configured to place a surface model comprising surface elements within the image, to determine for each surface element a respective sub volume ( 6 ) of the image and to use a neural network ( 51 ) for determining respective distances between the surface elements and the boundary of the object in the image based on the determined subvolumes. The surface model is then adapted in accordance with the determined distances, in order to segment the object. This segmentation, which is based on the subvolumes of the image and the neural network, is improved in comparison to known techniques which rely, ( 10 ) for instance, on a sampling of candidate points along lines being perpendicular to the respective surface element and on a determination of likelihoods for the candidate points that they indicate a boundary of the object.

FIELD OF THE INVENTION

The invention relates to a segmentation system, method and computerprogram for segmenting an object in an image. The invention furtherrelates to a training system, method and computer program for training aneural network.

BACKGROUND OF THE INVENTION

The article “Segmentation of the heart and great vessels in CT imagesusing a model-based adaptation framework” by O. Ecabert et al., MedicalImage Analysis, volume 15, pages 863 876 (2011) discloses a model-basedsegmentation technique for segmenting a heart in an image. Themodel-based segmentation includes placing a surface model of the heartin the image, wherein the surface model comprises surface elements beingtriangles. For each surface element candidate points are sampled along aline perpendicular to the respective surface element and for each ofthese candidate points the likelihood is determined that it correspondsto a boundary of the heart, wherein then the candidate point with thehighest likelihood is selected and the position of the respectivesurface element is adapted accordingly. For the selection of thecandidate point with the highest likelihood triangle-specific acceptancecriteria can be used like the criteria disclosed in the article“Optimizing boundary detection via Simulated Search with applications tomulti-modal heart segmentation” by J. Peters et al., Medical ImageAnalysis, volume 14, pages 70-84 (2010). This model-based segmentationdoes not always provide good segmentation results.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a segmentationsystem, method and computer program which allow for an improvedsegmentation of an object in an image. It is a further object of thepresent invention to provide a training system, method and computerprogram for training a neural network, which can be used for providingthe improved segmentation.

In a first aspect of the present invention a segmentation system forsegmenting an object in an image is presented, wherein the segmentationsystem comprises:

-   -   an image providing unit for providing an image of an object, the        image representing an image volume,    -   a model providing unit for providing a deformable surface model        for being adapted to a surface of the object, wherein the        surface model comprises surface elements defining respective        parts of a mesh surface,    -   a subvolumes determination unit for placing the surface model        within the image and for determining for each surface element of        the surface model a respective subvolume of the image such that        the respective subvolume overlaps with the respective surface        element,    -   a neural network providing unit for providing a convolutional        neural network being adapted to determine distances between        surface elements of the surface model and a boundary of the        object in the image based on determined subvolumes,    -   a distance determination unit for determining respective        distances between the surface elements of the provided and        placed surface model and the boundary of the object in the        provided image by using the provided neural network based on the        determined subvolumes, and    -   a model adaptation unit for adapting the surface model to the        object in the image in accordance with the determined distances.

By determining for a respective surface element a respective subvolume,by using the subvolumes together with the convolutional neural networkfor determining the distances between the surface elements and theboundary of the object in the image and by then using these distancesfor adapting the surface model, the segmentation of the object in theimage is improved in comparison to known techniques which rely, forinstance, on a sampling of candidate points along lines beingperpendicular to the respective surface element and on a determinationof likelihoods for the candidate points that they indicate a boundary ofthe object.

The image providing unit can be a storing unit, in which the image ofthe object is stored, and from which the image can be retrieved forproviding the same. The image providing unit can also be a receivingunit for receiving an image of the object and for providing the receivedimage. Moreover, the image providing unit can be an image generationunit being adapted to determine the image based on raw data of theobject. The image providing unit can also be an entire imaging systembeing adapted to acquire the raw data and determine, in particular,reconstruct, the image based on the raw data.

In an embodiment the neural network providing unit is adapted to providea further convolutional neural network being adapted to determineconfidence values for surface elements of the surface model based on thesubvolumes, wherein a confidence value for a respective surface elementis indicative of an estimation of a deviation of the distance determinedfor the respective surface element from the actual distance of therespective surface element to the boundary of the object in the image,wherein the segmentation system further comprises a confidence valuedetermination unit for determining confidence values for the surfaceelements of the surface model by using the provided further neuralnetwork based on the determined subvolumes. In particular, the modeladaptation unit is adapted to adapt the surface model in accordance withthe determined distances and based on the respective confidence valuedetermined for the respective surface element. For instance, the modeladaptation unit can be adapted to allow a distance determined for arespective surface element, for which a relatively large confidencevalue has been determined, to contribute stronger to the adaptationprocess than a distance determined for a respective surface element forwhich a relatively low confidence value has been determined.Accordingly, the degree of contribution to the adaptation process of adistance determined for a respective surface element may depend on theconfidence value determined for the respective surface element. This canlead to a further improved segmentation of the object in the image.

In an embodiment the neural network providing unit comprises severalconvolutional neural networks which are adapted to be used for differentsurface models representing different kinds of objects and/or fordifferent groups of surface elements of a same surface model, whereinthe neural network providing unit is adapted to provide, for determiningdistances for all surface elements of a surface model representing theobject or for a group of surface elements of the surface modelrepresenting the object, the corresponding neural network. Thus, fordifferent parts of a same object, different neural networks might beused for segmenting the object. Moreover, for segmenting the respectiveobject, a neural network might be used, which is specifically adaptedfor the class of objects, to which the object to be segmented belongsto. This can lead to a further improved determination of the distancesbetween the respective surface elements and the boundary of the objectin the image and hence finally to a further improved segmentation of theobject in the image.

The model providing unit is preferentially adapted to provide thesurface model such that each surface element comprises a direction,wherein the distance determination unit is adapted to determine therespective distance in the direction of the respective surface element.Moreover, preferentially the model providing unit is adapted to providea triangle mesh as the surface model, wherein the surface elements aretriangles. However, the surface model can also be another mesh, whereinthe surface elements can have another shape not being triangular. Theobject is preferentially an anatomical object like an organ or anotherpart of a human being or of an animal. However, the object can also be atechnical object.

In particular, the model providing unit provides the surface model suchthat for each surface element a respective three-dimensional orthogonalcoordinate system is defined, wherein one axis of the coordinate systemcoincides with the normal of the respective surface element. Thedirection of the axis, which is aligned with the normal, can bearbitrarily defined and defines the direction of the respective surfaceelement. For instance, if the surface element is a triangle having afirst vertex, a second vertex and a third vertex, the direction of theaxis, i.e. the orientation, might be defined by the cross product of a)a vector pointing to the second vertex minus a vector pointing to thefirst vertex and b) a vector pointing to the third vertex minus a vectorpointing to the first vertex.

The surface elements are preferentially planar and the two further axesare preferentially located in the respective plane of the respectivesurface element. For instance, the center of the coordinate system cancoincide with the center of the respective surface element, wherein thefirst axis is aligned with the normal, the second axis is within theplane of the respective surface element and points from the center toone vertex of the surface element and the third axis can be orthogonalto the first and second axes.

The subvolumes determination unit is preferentially adapted to determinethe subvolumes such that they are defined with respect to the coordinatesystem of the respective surface element. For instance, the respectivesubvolume can be rectangular having three axes, wherein one of theseaxes can coincide with the first axis of the coordinate system of therespective surface element being aligned with the normal of therespective surface element. Or, in another example, the respectivesubvolume can be cylindrical and the longitudinal axis of thecylindrical subvolume can coincide with the first axis of the coordinatesystem of the respective surface element. Since the respective subvolumeis preferentially defined with respect to the coordinate system of therespective surface element, the subvolumes are preferentiallyindependent of the pose and position of the object in the image.Correspondingly, also the input to the neural network is preferentiallyindependent of the pose and position of the object in the image, therebysimplifying the neural-network-based segmentation of the object in theimage.

The distance determination unit is preferentially adapted to determinesigned distances, wherein the sign is a sign with respect to the axis ofthe coordinate system of the respective surface element, which isaligned with the normal of the respective surface element. The signdefines in which direction the boundary of the object is present in theprovided image with respect to the surface element, i.e., for instance,whether the boundary is “above” or “below” the respective surfaceelement, or, in other words, in which direction the respective surfaceelement needs to be moved by the model adaptation unit, in order toadapt the surface model to the boundary of the object in the providedimage. For instance, it can be defined that, if the respective sign ispositive, the respective surface element needs to be moved in thedirection of the axis of the coordinate system being normal to therespective surface element and, if the respective sign is negative, therespective surface element needs to be moved in a direction beingopposite to a direction of the axis of the coordinate system beingnormal to the respective surface element.

In an embodiment the subvolumes determination unit is adapted todetermine the subvolumes such that they are elongated. Furthermore, themodel providing unit can be adapted to provide the surface model suchthat each surface element comprises, as explained above, a direction,wherein the subvolumes determination unit is adapted to determine thesubvolumes such that they are elongated in the direction of therespective surface element. In particular, the corresponding elongationdirection is preferentially aligned with the first axis of thecoordinate system of the respective surface element being aligned withits normal. However, in another embodiment the subvolumes can also benot non-elongated. For instance, they can be cubical or spherical.Moreover, it is preferred that the subvolumes determination unit isadapted to determine the subvolumes such that they have the samedimensions and the same shape. By using the same dimensions and the sameshape the computational efforts can be reduced.

In an embodiment the neural network providing unit is adapted to providethe convolutional neural network such that it is adapted to additionallydetermine further quantities being related to the image of the objectbased on the subvolumes of the image, wherein the distance determinationunit is also adapted to determine the further quantities by using theprovided neural network based on the determined subvolumes. Inparticular, the neural network providing unit and the distancedetermination unit can be adapted such that the further quantitiesinclude the normals of the boundary of the object in the image.

The model adaptation unit is adapted to adapt the surface model byconsidering the determined distances and optionally normals of theboundary of the object in the image. In particular, the model adaptationunit can be adapted to use an adaptation algorithm having a term in itscost function which tries to attract the respective surface element to arespective target point defined by the determined distance. Forinstance, the respective target point can be defined by the respectivedistance and the respective axis of the coordinate system being alignedwith the normal of the respective surface element, wherein the targetpoint is the point on the respective axis, which has the respectivedistance to the respective surface element. In a further embodiment themodel adaptation unit is adapted to use a cost function with a term thattries to attract the respective surface element not to the target point,but to a target plane approximating a tangent plane, wherein the targetplane is preferentially defined by the corresponding normal of theboundary of the object, i.e. the normal is perpendicular to the targetplane, and the respective distance. This attraction to the target planeand not to the target point allows the respective surface element to“slide” on the boundary of the object, which can lead to an improvedadaptation of the surface model to the object in the image. In general,the model adaptation unit is preferentially adapted to use a costfunction having different terms, wherein the adaptation process iscarried out by minimizing the cost function. The cost function comprisesa term which depends on the determined distances, wherein this term isreduced, if the distance between the surface elements and the boundariesare reduced. This term tries to, for instance, attract the respectivesurface element to a target point defined by the respective determineddistance or to a target plane defined by the respective determineddistance and normal. Furthermore, the model adaptation unit can beadapted to use an adaptation algorithm having a term in its costfunction which tries to orient the respective surface element such thatit is parallel to the target plane approximating the tangent plane. Thecost function can comprise further terms like a term which tries to keepa certain shape of the surface model. The terms of the cost functionmight also be regarded as being “energies”, i.e. one of these termsmight be regarded as being an internal energy and another of these termmight be regarded as being an external energy. For more detailsregarding the adaptation of the surface model reference is made to knownadaptation algorithms like the algorithms disclosed in the abovementioned articles by O. Ecabert et al. and J. Peters et al.

In an embodiment the neural network providing unit is adapted to providea single convolutional neural network being adapted to determinedistances to the boundary of the object for all surface elements of thesurface model, wherein the distance determination unit is adapted todetermine the respective distances between the surface elements of thesurface model and the boundary of the object in the image by using theprovided single neural network based on the determined subvolumes.Moreover, in an embodiment the image providing unit is adapted toprovide the image such that each image element comprises two or moreimage values. In particular, the image providing unit can be adapted toprovide the image such that the image values of a same image elementcorrespond to different imaging modalities or to different used imageacquisition protocols of a same imaging modality. If a single imageelement, i.e., for instance, a single voxel, comprises two or more imagevalues, the information of each image element is increased in comparisonto having only a single image value for a single image element. Thisincreased information provided by the image can lead to a furtherimproved determination of the distances between the surface elements ofthe surface model and the boundary of the object in the image,particularly if the different image values of a same image elementcorrespond to different imaging modalities which often emphasizedifferent aspects of a same object. In the case of providing an image,wherein a single image element has several image values, the neuralnetwork preferentially has also been trained with images having severalimage values in a single image element.

The subvolumes determination unit is preferentially adapted to determinethe subvolumes by sampling the image. In an embodiment the image issampled such that a degree of sampling depends on a distance from acenter of the respective subvolume. In particular, the image is sampledsuch that a degree of sampling decreases with increasing distance from acenter of the respective subvolume. For instance, if the subvolumes arerectangular, the image can be sampled such that a degree of samplingdecreases with increasing a distance from the normal of the respectivesurface element at the center of the respective subvolume in a directionbeing parallel to the respective surface element.

The subvolumes are preferentially rectangular. However, in an embodimentthey can be cylindrically shaped. In particular, the subvolumesdetermination unit can be adapted to determine the subvolumes bysampling the image such that a degree of sampling along a ring shapedpart of the cylinder depends on the radius of the ring. For instance,the sampling rate can be reduced with increasing distance to the centerof the respective surface element, wherein the resulting reduced overallsampling rate can lead to reduced computational efforts needed forsegmenting the object in the image. The reduction of the sampling ratecorresponds to an increase of gaps between samples. In an embodiment afixed number of samples is used for all rings such that with increasingdiameter naturally also the gap between the samples is increased, wherethe increase in gap directly relates to the distance of a sample fromthe center line of the cylinder. Alternatively, if the subvolumes arerectangular, rectangles, particularly squares, of increasing size in aplane being perpendicular to the respective surface element might beused while always sampling the same amount of sample points.Preferentially, for different surface elements the same sampling patternis used for determining the different subvolumes, wherein this samplingpattern might be a hexagonal pattern or another pattern. If thesubvolumes are cylindrical, the sampling pattern might be defined by twoor more rings, wherein the samples on neighboring rings are shifted withrespect to each other in a direction being orthogonal to the radius.

In an embodiment the segmentation system further comprises a trainingdata providing unit for providing a training image showing a trainingobject and for providing a deformable training surface model whichcomprises several surface elements and which has been adapted to thetraining object, and a training unit for a) determining several modifiedtraining surface models by modifying surface elements of the adaptedtraining surface model, b) determining subvolumes of the training imagefor the surface elements of the modified training surface models,wherein for a respective surface element a subvolume is determined,which overlaps the respective surface element, c) determining distancesfor the surface elements of the modified training surface models,wherein for a respective surface element a respective distance to theun-modified training surface model, which has been adapted to thetraining object in the training image, is determined, and d) trainingthe provided convolutional neural network based on determined subvolumesand determined distances. Thus, the segmentation system cannot only beadapted to segment the object, but also to train a new or an alreadytrained convolutional neural network for the segmentation procedure.

In a further aspect of the present invention a training system fortraining a neural network is presented, wherein the training systemcomprises:

-   -   a neural network providing unit for providing a convolutional        neural network,    -   a training data providing unit for providing a training image        showing a training object and for providing a deformable        training surface model which comprises several surface elements        and which has been adapted to the training object,    -   a training unit for training the provided neural network,        wherein the training unit is adapted to:    -   a) determine several modified training surface models by        modifying surface elements of the adapted training surface        model,    -   b) determine subvolumes of the training image for the surface        elements of the modified training surface models, wherein for a        respective surface element a subvolume is determined, which        overlaps the respective surface element,    -   c) determine actual distances for the surface elements of the        modified training surface models, wherein for a respective        surface element a respective distance to the un-modified        training surface model, which has been adapted to the training        object in the training image, is determined,    -   d) train the provided convolutional neural network based on the        determined subvolumes and the determined actual distances.

The training data providing unit is preferentially a storing unit inwhich the training image and the adapted training surface model arestored and from which the training image and the adapted trainingsurface model can be retrieved for providing the same. However, thetraining data providing unit can also be a receiving unit for receivingthe training image and the adapted training surface model from anotherunit and for providing the received training image and adapted trainingsurface model.

Preferentially the training unit is adapted to displace a surfaceelement and/or tilt a surface element for modifying the surface element.In this way several modified training surface models, which can be usedfor the training, can be determined in a relatively simple way. Inparticular, an unlimited amount of modified training surface modelshaving known distances to the boundary of the training object can begenerated for the training procedure as ground truth, in order to trainthe neural network as extensive as desired. This can lead to a very welltrained neural network and hence to a very exact segmentation, if thisneural network is used for segmenting an object in an image.

In an embodiment the training unit is adapted to a) determine simulateddistances for the surface elements of the modified training surfacemodels based on the determined corresponding subvolumes and the trainedconvolutional neural network and b) determine deviation values for thesurface elements of the modified training surface models, wherein for arespective surface element a respective deviation value is determined,which is indicative of a deviation of the respective simulated distancefrom the respective actual distance, wherein the neural networkproviding unit is adapted to provide for surface elements, for which therespective deviation value is larger than a threshold, a furtherconvolutional neural network, wherein the training unit is adapted totrain the provided further convolutional neural network based on thesubvolumes and the actual distances determined for the surface elementsfor which the respective deviation value is larger than the threshold.Thus, for different parts of the same object different neural networkcan be trained, wherein the respective neural network is optimized forthe respective part of the object. If the correspondingly trained neuralnetworks are used for segmenting an object in an image, the segmentationcan be further improved.

In an embodiment the training unit is adapted to a) determine simulateddistances for the surface elements of the modified training surfacemodels based on the determined corresponding subvolumes and the trainedconvolutional neural network, and b) determine confidence values for thesurface elements of the modified training surface models, wherein for arespective surface element a respective confidence value is determinedbased on a deviation of the respective simulated distance from therespective actual distance, wherein the neural network providing unit isadapted to provide a further convolutional neural network fordetermining confidence values for surface elements of a surface model ofan object based on the subvolumes, wherein the training unit is adaptedto train the further convolutional neural network based on theconfidence values and the subvolumes of the training image determinedfor the surface elements of the training surface model. Thus, thetraining can also result in a confidence neural network, which providesconfidence values during a segmentation process, wherein theseconfidence values can be used for further improving an adaptation of amodel surface to an object in an image and hence can lead to a furtherimproved segmentation. The confidence values preferentially depend onthe deviations such that the larger the deviation the smaller theconfidence values.

In another aspect of the present invention a segmentation method forsegmenting an object in an image is presented, wherein the segmentationmethod comprises:

-   -   providing an image of an object by an image providing unit, the        image representing an image volume,    -   providing a deformable surface model for being adapted to a        surface of the object, wherein the surface model comprises        several surface elements, by a model providing unit,    -   placing the surface model within the image and determining for        each surface element a respective subvolume of the image such        that the respective subvolume overlaps with the respective        surface element by a subvolumes determination unit,    -   providing a convolutional neural network being adapted to        determine distances between surface elements of the surface        model and a boundary of an object in an image based on        determined subvolumes by a neural network providing unit,    -   determining respective distances between the surface elements of        the provided and placed surface model and a boundary of the        object in the provided image by using the provided neural        network based on the determined subvolumes by a distance        determination unit, and    -   adapting the surface model in accordance with the determined        distances by a model adaptation unit.

In a further aspect of the present invention a training method fortraining a neural network is presented, wherein the training methodcomprises:

-   -   providing a convolutional neural network by a neural network        providing unit,    -   providing a training image showing a training object and        providing a deformable training surface model which, comprises        several surface elements and which has been adapted to the        training object, by a training data providing unit,    -   training the provided neural network by a training unit, wherein        the training includes:

a) determining several modified training surface models by modifyingsurface elements of the adapted training surface model,

-   -   b) determining subvolumes of the training image for the surface        elements of the modified training surface models, wherein for a        respective surface element a subvolume is determined, which        overlaps the respective surface element,    -   c) determining actual distances for the surface elements of the        modified training surface models, wherein for a respective        surface element a respective distance to the un-modified        training surface model, which has been adapted to the training        object in the training image, is determined,    -   d) training the provided convolutional neural network based on        the determined subvolumes and the determined actual distances.

In a further aspect of the present invention a segmentation computerprogram for segmenting an object in an image is presented, wherein thecomputer program comprises program code means for causing a segmentationsystem as defined in claim 1 to carry out the steps of the segmentationmethod as defined in claim 12, when the computer program is run on acomputer controlling the segmentation system.

In another aspect of the present invention a training computer programfor training a neural network is presented, wherein the computer programcomprises program code means for causing a training system as defined inclaim 10 to carry out the steps of the training method as defined inclaim 13, when the computer program is run on a computer controlling thetraining system.

It shall be understood that the segmentation system of claim 1, thetraining system of claim 10, the segmentation method of claim 12, thetraining method of claim 13, the segmentation computer program of claim14 and the training computer program of claim 15 have similar and/oridentical preferred embodiments, in particular, as defined in thedependent claims.

It shall be understood that a preferred embodiment of the presentinvention can also be any combination of the dependent claims or aboveembodiments with the respective independent claim.

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following drawings:

FIG. 1 shows schematically and exemplarily an embodiment of asegmentation system for segmenting an object in an image,

FIG. 2 illustrates the segmentation of the object in the image,

FIG. 3 shows schematically and exemplarily a triangular surface element,a corresponding subvolume and the distance to a boundary of the objectto be detected in the image,

FIG. 4 illustrates aspects of a neural network used by the segmentationsystem,

FIG. 5 shows schematically and exemplarily a training system fortraining a neural network,

FIGS. 6 and 7 illustrate schematically and exemplarily a generation of amodified training surface model,

FIG. 8 shows a flowchart exemplarily illustrating an embodiment of asegmentation method for segmenting an object in an image, and

FIG. 9 shows a flowchart exemplarily illustrating an embodiment of atraining method for training a neural network.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 shows schematically and exemplarily an embodiment of asegmentation system for segmenting an object in an image. In thisembodiment the object is a cortex of a human head and the image is amagnetic resonance (MR) image. The segmentation system comprises animage providing unit 2 for providing the image of the object, wherein inthis embodiment the image providing unit 2 is a storing unit in whichthe MR image of the cortex is stored and which is adapted to provide thestored MR image. The segmentation system 1 further comprises a modelproviding unit 4 for providing a deformable surface model (also referredto as ‘adaptable surface model’) for being adapted to a surface of thecortex, wherein the surface model comprises several surface elements. Inthis embodiment the surface model is a mesh of triangles, wherein thesurface elements are the triangles of the mesh. The segmentation system1 also comprises a subvolumes determination unit 5 for initially placingthe mesh within the MR image and for determining for each triangle ofthe mesh a respective subvolume of the MR image such that the respectivesubvolume overlaps with the respective triangle. Such initial placingmay, for example, comprise positioning the deformable surface model inthe image, for example at a default position or at the position of theobject in the image. In some embodiments, such initial placing may alsoinvolve globally scaling the deformable surface model to the object inthe image. In some embodiments, the initial placing may involve a globalregistration of the deformable surface model to the object in the image.Such initial placing typically does not involve locally deforming thesurface model to better fit the object in the image; the latter istypically a subsequent step, which in this specification and in thefield of image segmentation may also be referred to as ‘adapting’ or‘fitting’ of the deformable surface model.

In this embodiment each surface element, i.e. each triangle, comprises adirection being defined by the normal of the respective surface element,wherein the subvolumes are elongated and are determined such that theyare elongated in the direction of the respective surface element.

Moreover, the subvolumes determination unit 5 is adapted to determinethe subvolumes such that they all have the same dimensions and the sameshape. Furthermore, the subvolumes determination unit 5 ispreferentially adapted such that the center of the respective subvolumecoincides with the center of the respective surface element. In theupper left part of FIG. 2 some of the subvolumes 6 are schematically andexemplarily illustrated with respect to the surface model 3, i.e. withrespect to the mesh.

The segmentation system 1 further comprises a neural network providingunit 7 for providing a convolutional neural network being adapted todetermine distances between the surface elements of the surface modeland a boundary of the object in the image based on the determinedsubvolumes 6. This is illustrated in FIG. 3, where a subvolume 6 isshown, which has been determined for the triangle 8, wherein thedistance d between the center of the triangle 8 and the position, wherethe normal 12, which traverses the center of the triangle 8, meets theboundary 19 of the cortex in the provided image, is determined. Althoughin FIG. 3 the base of the subvolume 6 is smaller than the area of thetriangle 8, in another embodiment the subvolume might also have a basebeing larger than the area of the triangle, wherein in this case theintersection area of the respective subvolume with the respectivetriangle would be triangular and not rectangular as shown in FIG. 3.

Moreover, the segmentation system 1 comprises a distance determinationunit 9 for determining respective distances d between the surfaceelements 8 of the surface model 3 and the boundary 19 of the object inthe image by using the provided neural network based on the determinedsubvolumes 6. Thus, the subvolumes 6 are used as an input for the neuralnetwork, whereafter the neural network provides a respective distance dfor each surface element. In particular, the distance determination unit9 is adapted to collect all subvolumes 6 in a multi-dimensional array50, i.e. to collect all profiles 6 in a multi-dimensional array 50,wherein the provided convolutional neural network 51 is a fullyconvolutional neural network, i.e. all layers of the neural network areconvolutional layers, and wherein this fully convolutional neuralnetwork 51 is applied to the multi-dimensional array 50 for determiningthe distances d. This will in the following be explained in more detailwith reference to FIG. 2.

In FIG. 2 on the left hand side an illustration is given in twodimensions for simplicity. Meaning, the extracted subvolumes 6 are infact three-dimensional, although they are just drawn as two-dimensionalrectangles. In FIG. 2 on the right hand side more details about theactual architecture are illustrated and also the respective dimensionsof a three-dimensional array are shown and how those dimensions arecollapsed through the application of several convolutional layers inorder to yield a single one-dimensional column, where each valuecorresponds to a distance value of the respective triangle.

In FIG. 2 reference sign 59 indicates the number of voxels per slice ofa subvolume, wherein the term “slice” in this context refers to allvoxels of a subvolume that lie within a plane orthogonal to the trianglenormal, i.e. in this example the surface elements are triangles. Forcuboidal subvolumes, a slice is a rectangle, for cylindrical subvolumes,a slice would be a disc. Moreover, in FIG. 2 reference sign 54 indicatesthe number of triangles, i.e. the number of surface elements, of thesurface model and reference sign 55 refers to the number of slices persubvolume.

A natural way of collecting the voxel intensities of manythree-dimensional subvolumes would be in one four-dimensional array,where the first dimension of the array is the triangle index and theother three dimensions are the three dimensions of the three-dimensionalsubvolumes. However, since in this example only the dimension of thesubvolumes that is aligned with the triangle normal is of specialinterest, one dimension is removed by collecting all voxels of one slicein a one-dimensional vector. For example, the voxel values of a 40×5×5subvolume can be rearranged into a 40×25 subvolume by rearranging allvoxels of a 5×5 slice into a one-dimensional vector with 25 vectorelements. Thus, for a mesh, i.e. a surface model, consisting of, forinstance, 6000 triangles, the intensity values of the corresponding 6000three-dimensional subvolumes with dimension 40×5×5 can be collected inone three-dimensional array with the dimension 6000×50×25, where thefirst dimension indexes the respective triangle, the second dimensionindexes a slice within a three-dimensional subvolume and the lastdimension contains all voxels for a particular slice. This isrepresented by the three-dimensional array 50 illustrated in, forinstance, the upper right part of FIG. 2. Serializing all voxels withina slice has the added advantage of generalizing to differentwithin-slice sampling schemes. For example, points within a slice can besampled on concentric rings with increasing distance and then all samplepoints can be collected in a one-dimensional vector.

Preferentially, per triangle features are extracted, wherein thefeatures are learned as part of the convolutional neural network, whereeach convolutional layer of the network acts as a trainable featureextractor. An example of a network architecture will in the following beillustrated with reference to FIG. 4.

In this example it is assumed that the mesh, i.e. the surface model,consists of 5840 triangles and that for each triangle a subvolume ofdimension 40×5×5 is extracted. The voxel intensities are then collectedin a 5840×40×25 array, which in this example is the input into theneural network. The network is divided into blocks of convolutional,batch normalization, and rectified linear unit (ReLU) layers withdifferent kernel sizes, wherein in FIG. 4 the boxes represent operationsof layers of the neural network and wherein in each box with the label“CBR” operations of convolutional, batch normalization and ReLU layersare combined to simplify the drawing. Moreover, in FIG. 4 the box 85with the label “C” represents an operation of a convolutional layeronly. The first three layers 80, 81, 82 calculate a convolution with a1×7×25, 1×7×32, and 1×7×32 kernel, respectively. Since validconvolutions are used, the size of the three-dimensional array isreduced from 5840×40×25 to 5840×22×32. Moreover, since the firstdimension of the used kernels is 1, the sequence of convolutions doesnot change the size of the first dimension of the three-dimensionalarray. Thus, a response for each triangle is calculated, without losingsome triangles along the way. After the first three convolutions alldimensions of the three-dimensional array are still larger than one,wherein the number of elements in the second dimension has been reduced.This is indicated in FIG. 2 by the boxes 56 and 57. The first threeconvolutional layers 80, 81, 82 are followed by a convolutional layer 83with a kernel whose second dimension is equal to the second dimension ofits input. As a result, the dimension along the triangle normal of theinput is collapsed to 1. Consequently, the effective dimensionality ofthe three-dimensional volume is only 2, which in FIG. 2 is indicated bythe flat box 58 in the three-dimensional space. Similarly, the lastconvolutional layer 85 collapses the last dimension of the array andproduces a 5840×1×1 dimensional array indicated in FIG. 2 by referencenumber 59, only containing a single value per triangle, namely thepredicted distance of the respective triangle to the boundary of theobject. The box 84 represents an operation of another convolutionallayer with kernel size 1×1×32. Since this layer does not convolve overany axis other than the channel axis, it could also be seen as anadditional per-triangle dense layer and it serves the same purpose asregular dense layers do in classification/regression networks. Namely,it increases the non-linearity of the network and allows the network tolearn more complex dependencies between the input and the output of thenetwork. Other than that, there is nothing special about the operationof the layer represented by box 84 (unlike boxes 83 and 85 representingoperations of layers which also change the data layout).

The features that are used to estimate the distances are defined by theconvolutional kernels, which are automatically learned during thetraining procedure and preferentially tuned for a specific combinationof input modalities and target object. The features are defined by thelearned convolutional kernels and depend on the type of input modalityand target boundary. It would make sense for the network to learnedge-like feature in the first layers, but this behavior is nothard-coded and can vary depending on the input modality and type oftarget boundary.

The result of a convolutional layer is regarded as being a featureresponse, because calculating a convolution is regarded as a featureextraction step, where here the concept of a feature is rather abstract.The convolutional kernel is regarded as being a feature detector. In thehere exemplarily described network architecture, the responses of allfeature detectors are collected in the last dimension. A value atlocation (i,j,k) in box 56 or box 57 defines how strongly feature k ofthe respective layer responded at distance j for triangle i. Thosefeatures can be simple edge features that respond at a particularlocation for a particular triangle if an edge can be observed at thatlocation. Since these are features that are useful very early in thefeature extraction pipeline, they can be regarded as being low-levelfeatures. Box 58 does not contain feature responses for differentdisplacement candidates, but only feature responses of different filtersfor different triangles. Those features are necessarily much moreabstract than the simple edge features of the first layers. Therefore,they can be regarded as being high-level features. The determination ofthe high-level features depending on the low-level features can beregarded as being integration, wherein the integration procedure islearned by the neural network, wherein each high-level feature is acombination of low-level features of different displacement candidates.How those features are combined is defined by the convolutional kernelwhich is learned during training.

The concept described in this example is similar to the integration ofdisplacement candidates, where the collapse of the displacement axisforces the network to learn to integrate different displacementcandidates. Likewise, the last layer collapses the feature dimension,which forces the network to learn to integrate many features pertriangle into a single response. Since this response is trained with theobjective to minimize the difference of the predicted value with theknown distance of the triangle with the boundary, the network is forcedto learn object distances and the last layer in particular is forced tolearn to integrate features into distance estimates.

It should be noted that there is no concept in this architecture thatexplicitly calculates distances. The reason why the neural networkfinally outputs distances d for different triangles s (cf. curve 60 inFIG. 2) is because the network was trained to predict distances. Thatis, the output of the network depends more on the used objectivefunction than the actual architecture of the network. The finallydetermined distances 60 are also schematically shown overlaid over theinput array of subvolumes in the lower left part of FIG. 2 and indicatedthere by reference sign “52”.

The segmentation system 1 further comprises a model adaptation unit 10for adapting the surface model 3 in accordance with the determineddistances d. For this adaptation process known adaptation algorithms canbe used like the adaptation algorithm disclosed in the above mentionedarticles by O. Ecabert et al. and by J. Peters et al. These algorithmsor other known adaptation algorithms can be used especially to modifythe mesh of triangles such that it is adapted to target points definedby the determined distances and the triangle normals.

In this embodiment the neural network providing unit 7 is adapted toprovide, for the determination of the distances for all surface elementsof the surface model, a single convolutional neural network, wherein thedistance determination unit 9 is adapted to determine the respectivedistances between the surface elements 8 of the surface model 3 and theboundary 19 of the object in the image by using the provided singleneural network based on the determined subvolumes. However, the neuralnetwork providing unit 7 can also be adapted to provide several neuralnetworks for different groups of surface elements of the same surfacemodel 3, wherein the distance determination unit 9 is then adapted tocollect for each neural network the subvolumes, which have beendetermined for the surface elements of the respective group of surfaceelements, in order to determine a respective multi-dimensional array 50of respective subvolumes, which is then used by the respective neuralnetwork for determining the distances of the surface elements of therespective group. The neural network providing unit 7 can also compriseseveral convolutional neural networks which are adapted to be used fordifferent surface models representing different kinds of objects,wherein the neural network providing unit 7 can be adapted to providethe respective neural network, which corresponds to the kind of theactual object to be segmented in the image.

The neural network providing unit 7 can also be adapted to provide afurther convolutional neural network being adapted to determineconfidence values for surface elements 8 of the surface model 3 based onthe subvolumes 6, wherein a confidence value for a respective surfaceelement 8 is indicative of an estimation of a deviation of the distancedetermined for the respective surface element 8 from the actual distanceof the respective surface element 8 to the boundary 19 of the object inthe image, wherein the confidence value is higher if the estimateddeviation is smaller. The segmentation system 1 then further comprises aconfidence value determination unit 11 for determining confidence valuesfor the surface elements 8 of the surface model 3 by using the providedfurther neural network based on the determined subvolumes 6. The modeladaptation unit 10 can then be adapted to adapt the surface model 3 inaccordance with the determined distances d such that during theadaptation a degree of consideration of a respective distance determinedfor a respective surface element 8 depends on the respective confidencevalue determined for the respective surface element 8. For instance,while using known model adaptation algorithms like the algorithmsdisclosed in the above mentioned articles by O. Ecabert et al. and by J.Peters et al., which adapt a surface model to an object based ondetermined distances, for a respective surface element an attraction ofthis surface element to a target point defined by the respectivedetermined distance might be disabled, if for this surface element theconfidence value was too low, i.e., for instance, smaller than apredefined threshold. Also a weighting of this attraction part of theadaptation algorithm depending on the respective confidence value mightbe used.

The neural network providing unit 7 may be adapted to provide theconvolutional neural network such that it is adapted to additionallydetermine further quantities being related to the image of the objectbased on the subvolumes of the image, wherein the distance determinationunit 9 can be adapted to also determine the further quantities by usingthe provided neural network based on the determined subvolumes. Inparticular, the neural network providing unit 7 and the distancedetermination unit 9 can be adapted such that the further quantitiesinclude a normal of the boundary of the object in the image. Moreover,also the neural network used for determining the distances can beadapted to provide a confidence score, which might also be regarded asbeing a reliability of the detected boundary, for a respectivedetermined distance.

In an embodiment the distance determination unit or another unit of thesegmentation system is adapted to aggregate the confidence scoredetermined for all surface elements or for a part of the surfaceelements, in order to measure the reliability of the overallsegmentation or of a part of the segmentation, respectively. If thesegmentation is not reliable enough, a user might be informedaccordingly. The distance determination unit or another unit of thesegmentation system can also be adapted to use the determined confidencescore for identifying regions of the surface model in which theadaptation and hence the segmentation is not reliable enough. Forinstance, the confidence scores of the surface elements of a region canbe averaged and the resulting average confidence score can be comparedwith a predefined threshold, in order to determine whether thesegmentation was reliable enough for this region. The result can beoutput. For example, a region, for which a segmentation was not reliableenough, could be highlighted on a visualization of the adapted surfacemodel on a display 16.

The segmentation system 1 further comprises a training data providingunit 13 for providing a training image showing a training object and forproviding a deformable training surface model, which comprises severalsurface elements and which has been adapted to the training object.Moreover, the segmentation unit can comprise a training unit 14 for a)determining several modified training surface models by modifyingsurface elements of the adapted training surface model, b) determiningsubvolumes of the training image for the surface elements of themodified training surface models, wherein for a respective surfaceelement a subvolume is determined, which overlaps the respective surfaceelement, c) determining distances for the surface elements of themodified training surface models, wherein for a respective surfaceelement, a respective distance to the un-modified training surfacemodel, which has been adapted to the training object in the trainingimage, is determined, and d) training the provided convolutional neuralnetwork based on the determined subvolumes and the determined distances.Thus, the segmentation system can also be adapted to train a newconvolutional neural network or to further train an already trainedconvolutional neural network. However, the training of the neuralnetwork can also be carried out by a dedicated training system whichwill be described further below. Also details of the training will bedescribed further below.

The segmentation system 1 also comprises an input unit 15 like akeyboard, a computer mouse, a touchpad et cetera and the display 16 forshowing, for instance, the surface model adapted to the object in theimage.

FIG. 5 shows schematically and exemplarily an embodiment of a trainingsystem for training a neural network. The training system 30 comprises aneural network providing unit 37 for providing a convolutional neuralnetwork, wherein in this embodiment the neural network providing unit 37is a storing unit in which the neural network is stored and which isadapted to provide the stored convolutional neural network. The providedconvolutional neural network can be an untrained neural network or aneural network which has already been trained and which should befurther trained.

The training system 30 further comprises a training data providing unit13 for providing a training image showing a training object and forproviding an adapted training surface model which comprises severalsurface elements and which has been adapted to the training object.Moreover, the training system 30 comprises a training unit 14 fortraining the provided neural network, wherein the training unit 14 isadapted to determine several modified training surface models bymodifying surface elements of the adapted training surface model and todetermine subvolumes of the training image for the surface elements ofthe modified training surface models, wherein for a respective surfaceelement a respective subvolume is determined, which overlaps therespective surface element. This modification of the surface elementsand this determination of the subvolumes are schematically andexemplarily illustrated in FIGS. 5 and 6.

FIGS. 5 and 6 show a cortex 33 of a human head 39 as an example of atraining object, wherein in FIG. 6 the determined subvolumes 36 havebeen determined for surface elements of the training surface model,which have been displaced in the normal directions 32. In FIG. 7 it isillustrated that the surface elements and hence the correspondingdetermined subvolumes 36 can also be modified by tilting the respectivesurface elements of the training surface model.

The training unit 14 is further adapted to determine actual distancesfor the surface elements of the modified training surface models, i.e.of the displaced and/or tilted surface elements, wherein for arespective surface element a respective distance to the un-modifiedtraining surface model, which has been adapted to the training object inthe training image and which has been provided by the training dataproviding unit 13, is determined. Moreover, the training unit 14 isadapted to train the convolutional neural network, which has beenprovided by the neural network providing unit 37, based on thedetermined subvolumes 36 and the determined actual distances. Thus, theprovided convolutional neural network can be trained such that, giventhe determined subvolumes, deviations between the determined actualdistances and the distances output by the convolutional neural networkare minimized. This training can be carried out iteratively, wherein ineach iteration step the convolutional neural network is trained withanother set of subvolumes 36 which have been determined based onmodified surface elements of a respective modified training surfacemodel. In other words, in each iteration step another modified trainingsurface model might be used for the training.

The training unit 14 can be further adapted to determine simulateddistances for the surface elements of the modified training surfacemodels based on the determined corresponding subvolumes 36 and thetrained convolutional neural network and to determine deviation valuesfor the surface elements of the modified training surface models,wherein for a respective surface element a respective deviation value isdetermined, which is indicative of a deviation of the respectivesimulated distance from the respective actual distance, wherein theneural network providing unit 37 is adapted to provide for surfaceelements, for which the respective deviation value is larger than athreshold, a further convolutional neural network, wherein the trainingunit 14 is adapted to train the provided further convolutional neuralnetwork based on the subvolumes and the actual distances determined forthe surface elements for which the respective deviation value is largerthan the threshold. Thus, for different parts of the training surfacemodel different neural networks can be trained such that later, i.e.during an actual object segmentation procedure, for different parts ofan object to be segmented in an image, different convolutional neuralnetworks can be used as explained above.

The training unit 14 can also be adapted to determine simulateddistances for the surface elements of the modified training surfacemodels based on the determined corresponding subvolumes 36 and thetrained convolutional neural network and to determine confidence valuesfor the surface elements of the modified training surface models,wherein for a respective surface element a respective confidence valueis determined based on the deviation of the respective simulateddistance from the respective actual distance. The neural networkproviding unit 37 is then adapted to provide a further convolutionalneural network for determining confidence values for surface elements ofa surface model of an object based on the subvolumes, wherein thetraining unit 14 is adapted to train the further convolutional neuralnetwork based on the confidence values and the subvolumes of thetraining image determined for the surface elements of the trainingsurface model. This further convolutional neural network, which might beregarded as being a confidence convolutional neural network, can be usedduring a segmentation of an object in an image as explained above.

In the following an embodiment of a segmentation method for segmentingan object in an image will exemplarily be described with reference to aflowchart shown in FIG. 8.

In step 101 an image of an object is provided by the image providingunit 2. For instance, an MR image of a cortex is provided as the image.In step 102 a deformable surface model for being adapted to a surface ofthe object is provided by the model providing unit 4, wherein thesurface model comprises several surface elements. For instance, adeformable triangle mesh of a cortex is provided by the model providingunit 4. Moreover, in step 103 a convolutional neural network isprovided, which is adapted to determine distances between surfaceelements of the surface model and a boundary of the object in the imagebased on subvolumes of the provided image, wherein the neural networkproviding unit 7 provides this convolutional neural network. Theprovided convolutional neural network has been trained by the trainingsystem 30.

In step 104 the provided surface model is placed within the providedimage and for each surface element of the surface model a respectivesubvolume of the image is determined such that the respective subvolumeoverlaps with the respective surface element, wherein this arranging andthis determining is carried out by the subvolumes determination unit 5.In step 105 the distance determination unit 9 determines respectivedistances between the surface elements of the surface model and theboundary of the object in the image by using the provided neural networkbased on the determined subvolumes, and in step 106 the provided surfacemodel is adapted in accordance with the determined distances by themodel adaptation unit 10, in order to segment the object in the image.

In the following an embodiment of a training method for training aneural network will exemplarily be described with reference to aflowchart shown in FIG. 9.

In step 201 a convolutional neural network is provided by the neuralnetwork providing unit 37 and in step 202 a training image showing atraining object and a deformable training surface model are provided bythe training data providing unit 13, wherein the provided trainingsurface model comprises several surface elements and has been adapted tothe training object. In step 203 a modified training surface model isdetermined by the training unit 14 by modifying surface elements of theadapted training surface model. In particular, surface elements of theadapted training surface model are displaced and/or tilted randomly orby known amounts, in order to determine the modified training surfacemodel. In step 204 subvolumes of the training image are determined bythe training unit 14 for the surface elements of the modified trainingsurface model, wherein for a respective surface element a subvolume isdetermined, which overlaps the respective surface element. Inparticular, for each modified surface element an elongated subvolume isdetermined, wherein the elongation direction of the respective subvolumeis aligned with a normal of the respective surface element and whereinall determined subvolumes have the same shape and the same dimensions.In step 205 actual distances are determined for the surface elements ofthe modified training surface models, wherein for a respective surfaceelement a respective distance to the unmodified training surface model,which had been adapted to the training object in the training image andwhich had been provided by the training data providing unit 13 in step202, are determined. This determination can be carried out byconsidering the positions and orientations of the unmodified surfaceelements of the unmodified training surface model provided in step 202and the positions and orientations of the respective modified surfaceelements of the modified training surface model. However, thisdetermination of the actual distances can also just be using thedisplacements and/or tilts of the surface elements, if they have beendisplaced and/or tilted in known amounts in step 203. In step 206 theprovided convolutional neural network is trained based on the determinedsubvolumes and the determined actual distances.

In step 207 it is checked whether an abort criterion is fulfilled. Forinstance, it is checked whether the convolutional neural network hadbeen trained by a desired number of modified training surface models. Ifthis criterion is fulfilled, the method ends in step 208. Otherwise, themethod continues with step 203, wherein a further modified trainingsurface model is determined by modifying the surface elements of theadapted training surface model provided in step 202 or by modifyingsurface elements of an already modified training surface model.

In known model segmentation procedures, which might also be regarded asbeing boundary detection approaches, in the context of model adaptationthe task is normally broken into two steps, a sampling candidate pointsalong a line perpendicular to a mesh element like a triangle and aselection of a most suitable candidate point on the line using aclassifier selected via, for instance, Simulated Search disclosed in theabove mentioned article by J. Peters et al. Although this two-stepapproach can yield good segmentation results, it has been found that thereliability and accuracy of boundary detection can be increased if thesearch step and the classification step are integrated and the distanceto the desired boundary of the object in the image is directly estimatedfrom image values, especially gray values, surrounding a mesh triangleusing a learning approach, particularly an end-to-end-machine learningapproach. This increased reliability and accuracy of boundary detectiondirectly translates into improved reliability and accuracy of modeladaptation and hence segmentation of the object in the image. Thus, nosample points are involved any more, but a continuously valued distancecan be estimated by the above described segmentation system and method,wherein in an embodiment an oriented subvolume, which is preferentiallycentered at the respective mesh surface, i.e. at the respective surfaceelement, and which is oriented according to the local coordinate systemencoded in the respective mesh, is mapped to a scalar distance value byusing the convolutional neural network.

The neural network providing unit is preferentially adapted to provide aneural network which provides a real-valued outcome. Thus, the providedneural network is preferentially not a classification neural networkproviding a categorical outcome. Correspondingly, the provided neuralnetwork preferentially does not comprise a softmax layer as the lastlayer and is preferentially not trained to output a probabilitydistribution. It can be trained to output continuous values and could beregarded as being adapted for regression, in contrast to being adaptedfor classification.

The segmentation system and method are therefore adapted to extractimage subvolumes on the model surface, wherein, for instance, for eachtriangle in the case of a triangle mesh an orientation of a subvolumemay be derived from a respective coordinate system of the respectivetriangle. The segmentation system and method can be further adapted toestimate with one or several trained convolutional neural networks thedisplacement of the respective subvolume center, i.e., for instance, themesh point where the subvolume is centered, with respect to the desiredmodel surface, i.e. the boundary of the object in the image. Theestimated displacement information is subsequently used to adapt thesurface model to the object in the image.

The training unit is preferentially adapted to train the one or severalconvolutional neural networks by using deep learning, particularly deepend-to-end learning, wherein a ground truth segmentation is used for atraining image or several ground truth segmentations are used forseveral training images. Surface parts, i.e. the surface elements, arepreferentially displaced and/or tilted, subvolumes are extracted, andpreferentially deep learning is used to learn the displacement, i.e. thedistances, from the ground truth segmentations.

Although in above described embodiments the surface model is a trianglemesh, in other embodiments the surface model can also be a non-trianglemesh, i.e. a mesh whose surface elements are not triangular.

Image subvolumes, which are preferentially elongated, are extractedpreferentially for all surface elements, particularly for all triangles,of the actual surface model, wherein preferentially all subvolumes havethe same dimensions and the same shape and wherein the coordinate systemof the subvolumes is preferentially defined via the respectivecoordinate systems of the respective triangles. In particular, the axisof elongation of a respective image subvolume is parallel to arespective triangle normal. Preferentially all subvolumes are collectedin a multi-dimensional array and a fully convolutional neural network isused to estimate the displacement of each image subvolume, i.e. of eachsurface element, with respect to the desired boundary of the object inthe image.

The training system and method can be adapted to, for instance, displaceand/or tilt the surface elements by known amounts for generating themodified training surface models or they can be randomly displacedand/or tilted during each training iteration, wherein the training ofthe convolutional neural network preferentially relates to an updatingof the weights of the convolutional neural network on the basis of thesimulated displacements and/or tilts using back propagation.

Preferentially, the average root mean square distance between thesimulated and the predicted displacements is used as training criterion,i.e. the convolutional neural network is trained such that this averageroot mean square distance or another measure for a deviation between theoutput of the convolutional neural network, i.e. the simulated orestimated distances, and the displacements, i.e. the actual distances,is minimized.

As explained above, the training system and hence also the trainingmethod may be adapted to, instead of training one global convolutionalneural network that is used for all surface elements, particularlytriangles, of the surface model, train separate convolutional neuralnetworks to more specifically characterize different parts of the objectand its appearance in the image. The object is preferentially ananatomical object and the image is preferentially a medical image. Forinstance, in the field of radiation therapy planning bladder and bonesmay have fundamentally different appearances like it is the case in MRor computed tomography (CT) images and might benefit from using twodifferent neural networks for boundary detection. The number of separateneural networks and their associations with particular mesh triangles,i.e. with particular surface elements, can be either predetermined,i.e., for instance, different networks for different organs or organstructures, or learned during training. In particular, a subset oftriangles associated with a high boundary detection error after initialtraining may be selected and used to train a second neural network inorder to further improve boundary detection for these triangles.Furthermore, from the simulated errors observed during or after trainingthe neural network an extra neural network may be trained to provideconfidence scores, i.e. the confidence values, that can be used toincrease or decrease the external energy, i.e. the “image force”,associated with a detected boundary.

Although in above described embodiments the image comprises imageelements, wherein each image element comprises a respective single imagevalue, in other embodiments the respective image element can comprisetwo or more image values. Thus, for instance, the image may be amulti-protocol or multi-channel image with two or more intensity valuesassociated with each image element. Accordingly, the convolutionalneural network may be trained on subvolumes containing an n-tuple ofintensity values for each image element being preferentially a voxel. Amulti-protocol image is a combination of several images which have beenacquired using different protocols, wherein to a respective positionwithin the image the corresponding image values of the several imagesare assigned. The different protocols can refer to different imagingmodalities, i.e., for instance, a first image can be a computedtomography image and a second image can be a magnetic resonance image.The different protocols can also refer to different image acquisitionparameters of a same imaging modality for generating different images.The generation of the input array to the network is preferentiallysimilar to the uni-channel or uni-protocol case, i.e. to the case whereto each voxel only a single voxel value is assigned. For instance, foreach protocol subvolumes can be extracted, i.e. determined, for eachtriangle. For each subvolume, all voxels of one slice can be serializedinto a one-dimensional vector, wherein, in order to collect data frommultiple protocols, those one-dimensional vectors can be concatenated toyield one vector that contains all intensity values of all images for aparticular triangle, wherein this vector together with the correspondingother vectors of all subvolumes can be used as input for the neuralnetwork which then determines the distances. For example, if a mesh with5000 triangles is used, a subvolume size of 5×5×40, and two inputmodalities (e.g., T1-weighted and T2-weighted magnetic resonanceimaging) are considered, the input to the neural network could be a5000×40×(5*5*2), i.e. 5000×40×50, array.

Preferentially the subvolumes are sampled on a regular grid. However, inorder to reduce the number of samples that need to be processed, thesubvolumes may be sampled in rings around the center line, i.e. the linethat is perpendicular to the surface element and passes through thecenter of the surface element, wherein more distant rings are sampledmore sparsely. Thus, the subvolumes can be cylindrically shaped.

The provided images can be two-dimensional images, three-dimensionalimages or four-dimensional images, i.e. they can also depend on thetime.

Although in above described embodiments the images are CT images or MRimages, they can also be images of another imaging modality likeultrasound images.

Although in above described embodiments all subvolumes of all surfaceelements are used as input for the neural network for determining thedistances and/or for training the network, in other embodiments also asingle subvolume can be used as an input for the neural network and theneural network can be trained for providing a single distance for thesingle subvolume and possibly further parameters for the singlesubvolume like a normal of the boundary of the object in the image.However, using all subvolumes as input is preferred, because it can leadto better segmentation results. For instance, this may allow the neuralnetwork to learn shared weights during training, that is, to enforce forall triangles the same set of weights are learned for predicting theboundary.

The segmentation system and/or the training system may be embodied as,or in, a single device or apparatus, such as a workstation or imagingapparatus or mobile device. The device or apparatus may comprise one ormore microprocessors which execute appropriate software. The softwaremay have been downloaded and/or stored in a corresponding memory, e.g.,a volatile memory such as RAM or a non-volatile memory such as Flash.Alternatively, the functional units of the system may be implemented inthe device or apparatus in the form of programmable logic, e.g., as aField-Programmable Gate Array (FPGA). In general, each functional unitof the system may be implemented in the form of a circuit. It is notedthat each system may also be implemented in a distributed manner, e.g.,involving different devices or apparatuses. For example, thedistribution may be in accordance with a client-server model, e.g.,using a server and a thin-client.

Here and elsewhere, any ‘providing unit’, such as the image providingunit, the model providing unit, the neural network providing unit or thetraining data providing unit, may be embodied as an input interface foracessing the respective data. The input interface may take variousforms, such as a network interface to a Local Area Network (LAN) or aWide Area Network (WAN), such as the Internet, a storage interface to aninternal or external data storage, e.g., a volatile or non-volatilememory, harddisk, solid state storage, etc. The image providing unit maythus be embodied as an image input interface. The model providing unitmay thus be embodied as an model data input interface. The neuralnetwork providing unit may thus be embodied as an neural network datainput interface. The training data providing unit may thus be embodiedas a training data input interface. Two or more of such ‘providingunits’ may be embodied as a single input interface.

Units such as the subvolumes determination unit, the neural networkproviding unit, the distance determination unit and the model adaptationunit may be implemented by a processor, or a system of processors, whichis/are configured by suitable software to perform the describedfunctions. For example, the segmentation system may comprise a processorconfigured to internally communicate with the input interface(s) and amemory accessible by the processor. The memory may store instructions tocause the processor to perform the functions as described elsewhere inrelation to the subvolumes determination unit, the neural networkproviding unit, the distance determination unit and/or the modeladaptation unit.

Likewise, in the training system, units such as the training unit may beimplemented by a processor, or a system of processors, which is/areconfigured by suitable software to perform the described functions. Forexample, the training system may comprise a processor configured tointernally communicate with the input interface(s) and a memoryaccessible by the processor. The memory may store instructions to causethe processor to perform the functions as described elsewhere inrelation to the training unit.

Each method described in this specification may be implemented on acomputer as a computer implemented method, as dedicated hardware, or asa combination of both. Instructions for the computer, e.g., executablecode, may be stored on a computer readable medium, e.g., in the form ofa series of machine-readable physical marks and/or as a series ofelements having different electrical, e.g., magnetic, or opticalproperties or values. The executable code may be stored in a transitoryor non-transitory manner. Examples of computer readable mediums includememory devices, optical storage devices, integrated circuits, servers,online software, etc.

Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims.

In the claims, the word “comprising” does not exclude other elements orsteps, and the indefinite article “a” or “an” does not exclude aplurality.

A single unit or device may fulfill the functions of several itemsrecited in the claims. The mere fact that certain measures are recitedin mutually different dependent claims does not indicate that acombination of these measures cannot be used to advantage.

Procedures like the provision of the image, the provision of thedeformable surface model, the provision of the convolutional neuralnetwork, the determination of the subvolumes, the determination of thedistances, the adaptation of the model, et cetera performed by one orseveral units or devices can be performed by any other number of unitsor devices. These procedures and/or the control of the segmentationsystem in accordance with the segmentation method and/or the control ofthe training system in accordance with the training method can beimplemented as program code means of a computer program and/or asdedicated hardware.

A computer program may be stored/distributed on a suitable medium, suchas an optical storage medium or a solid-state medium, supplied togetherwith or as part of other hardware, but may also be distributed in otherforms, such as via the Internet or other wired or wirelesstelecommunication systems.

Any reference signs in the claims should not be construed as limitingthe scope.

The invention relates to a segmentation system for segmenting an objectin an image. The segmentation system is configured to place a surfacemodel comprising surface elements within the image, to determine foreach surface element a respective subvolume of the image and to use aneural network for determining respective distances between the surfaceelements and the boundary of the object in the image based on thedetermined subvolumes. The surface model is then adapted in accordancewith the determined distances, in order to segment the object. Thissegmentation, which is based on the subvolumes of the image and theneural network, is improved in comparison to known techniques whichrely, for instance, on a sampling of candidate points along lines beingperpendicular to the respective surface element and on a determinationof likelihoods for the candidate points that they indicate a boundary ofthe object.

1. A segmentation system for segmenting an object in an image, thesegmentation system comprising: an image providing unit for providing animage of an object, the image representing an image volume, a modelproviding unit for providing a deformable surface model for beingadapted to a surface of the object, wherein the surface model comprisessurface elements defining respective parts of a mesh surface, asubvolumes determination unit for placing the surface model within theimage and for determining for each surface element of the surface modela respective subvolume of the image such that the respective subvolumeoverlaps with the respective surface element, a neural network providingunit for providing a convolutional neural network being adapted todetermine distances between surface elements of the surface model and aboundary of the object in the image based on determined subvolumes, adistance determination unit for determining respective distances betweenthe surface elements of the provided and placed surface model and theboundary of the object in the provided image by using the providedneural network based on the determined subvolumes, and a modeladaptation unit for adapting the surface model to the object in theimage in accordance with the determined distances.
 2. The segmentationsystem as defined in claim 1, wherein the neural network providing unitis adapted to provide a further convolutional neural network beingadapted to determine confidence values for surface elements of thesurface model based on the subvolumes, wherein a confidence value for arespective surface element is indicative of an estimation of a deviationof the distance determined for the respective surface element from theactual distance of the respective surface element to the boundary of theobject in the image; and wherein the segmentation system furthercomprises a confidence value determination unit for determiningconfidence values for the surface elements of the surface model by usingthe provided further neural network based on the determined subvolumes.3. The segmentation system as defined in claim 2, wherein the modeladaptation unit is adapted to adapt the surface model in accordance withthe determined distances and based on the respective confidence valuedetermined for the respective surface element.
 4. The segmentationsystem as defined in claim 1, wherein the model providing unit isadapted to provide the surface model such that each surface elementcomprises a direction aligned with the normal of the respective surfaceelement, wherein the distance determination unit is adapted to determinethe respective distance in the direction of the respective surfaceelement.
 5. The segmentation system as defined in claim 1, wherein themodel providing unit is adapted to provide the surface model such thateach surface element comprises a direction aligned with the normal ofthe respective surface element, wherein the subvolumes determinationunit is adapted to determine the subvolumes such that they are elongatedin the direction of the respective surface element.
 6. The segmentationsystem as defined in claim 1, wherein the subvolumes determination unitis adapted to determine the subvolumes such that they have the samedimensions and the same shape.
 7. The segmentation system as defined inclaim 1, wherein the image providing unit is adapted to provide theimage such that each image element comprises two or more image values,the two or more image values being obtained from different imagingmodalities or from different image acquisition protocols of a sameimaging modality.
 8. The segmentation system as defined in claim 1,wherein the subvolumes determination unit is adapted to determinecylindrically shaped subvolumes.
 9. The segmentation system as definedin claim 1, wherein the subvolumes determination unit is adapted todetermine the subvolumes by sampling the image, wherein the image issampled such that a degree of sampling depends on a distance from acenter of the respective subvolume.
 10. A training system for training aneural network, the training system comprising: a neural networkproviding unit for providing a convolutional neural network, a trainingdata providing unit for providing a training image showing a trainingobject and for providing a deformable training surface model whichcomprises surface elements defining respective parts of a mesh surface,wherein the surface model has been adapted to the training object, atraining unit for training the provided neural network, wherein thetraining unit is adapted to: (a) determine several modified trainingsurface models by modifying surface elements of the adapted trainingsurface model, (b) determine subvolumes of the training image for thesurface elements of the modified training surface models, wherein for arespective surface element a subvolume is determined, which overlaps therespective surface element, (c) determine actual distances for thesurface elements of the modified training surface models, wherein for arespective surface element a respective distance to the un-modifiedtraining surface model, which has been adapted to the training object inthe training image, is determined, and (d) train the providedconvolutional neural network based on the determined subvolumes and thedetermined actual distances.
 11. The training system as defined in claim10, wherein the training unit is adapted to: determine simulateddistances for the surface elements of the modified training surfacemodels based on the determined corresponding subvolumes and the trainedconvolutional neural network, and determine confidence values for thesurface elements of the modified training surface models, wherein for arespective surface element a respective confidence value is determinedbased on a deviation of the respective simulated distance from therespective actual distance, wherein the neural network providing unit isadapted to provide a further convolutional neural network fordetermining confidence values for surface elements of a surface model ofan object based on the subvolumes, wherein the training unit is adaptedto train the further convolutional neural network based on theconfidence values and the subvolumes of the training image determinedfor the surface elements of the training surface model.
 12. Asegmentation method for segmenting an object in an image, thesegmentation method comprising: providing an image of an object by animage providing unit, the image representing an image volume, providinga deformable surface model for being adapted to a surface of the object,wherein the surface model comprises surface elements defining respectiveparts of a mesh surface, by a model providing unit, placing the surfacemodel within the image and determining for each surface element of thesurface model a respective subvolume of the image such that therespective subvolume overlaps with the respective surface element by asubvolumes determination unit, providing a convolutional neural networkbeing adapted to determine distances between surface elements of thesurface model and a boundary of the object in the image based ondetermined subvolumes by a neural network providing unit, determiningrespective distances between the surface elements of the provided andplaced surface model and the boundary of the object in the providedimage by using the provided neural network based on the determinedsubvolumes by a distance determination unit, and adapting the surfacemodel to the object in the image in accordance with the determineddistances by a model adaptation unit.
 13. A training method for traininga neural network, the training method comprising: providing aconvolutional neural network by a neural network providing unit,providing a training image showing a training object and providing adeformable training surface model which, comprises surface elementsdefining respective parts of a mesh surface, wherein the surface modelhas been adapted to the training object, by a training data providingunit, and training the provided neural network by a training unit,wherein the training includes: (a) determining several modified trainingsurface models by modifying surface elements of the adapted trainingsurface model, (b) determining subvolumes of the training image for thesurface elements of the modified training surface models, wherein for arespective surface element a subvolume is determined, which overlaps therespective surface element, (c) determining actual distances for thesurface elements of the modified training surface models, wherein for arespective surface element a respective distance to the un-modifiedtraining surface model, which has been adapted to the training object inthe training image, is determined, and (d) training the providedconvolutional neural network based on the determined subvolumes and thedetermined actual distances.
 14. A non-transitory computer readablemedium storing instructions executable by an electronic data processingdevice, the non-transitory computer readable medium comprisinginstructions that, when executed, cause the data processing device to:provide an image of an object by an image providing unit, the imagerepresenting an image volume; provide a deformable surface model forbeing adapted to a surface of the object, wherein the surface modelcomprises surface elements defining respective parts of a mesh surface,by a model providing unit; place the surface model within the image anddetermining for each surface element of the surface model a respectivesubvolume of the image such that the respective subvolume overlaps withthe respective surface element by a subvolumes determination unit,provide a convolutional neural network being adapted to determinedistances between surface elements of the surface model and a boundaryof the object in the image based on determined subvolumes by a neuralnetwork providing unit; determine respective distances between thesurface elements of the provided and placed surface model and theboundary of the object in the provided image by using the providedneural network based on the determined subvolumes by a distancedetermination unit; and adapt the surface model to the object in theimage in accordance with the determined distances (d) by a modeladaptation unit.
 15. A non-transitory storage medium storinginstructions executable by an electronic data processing device, thenon-transitory storage medium comprising instructions for training aconvolutional neural network, wherein the training includes: (a)determining several modified training surface models, wherein thetraining surface models comprise surface elements defining respectiveparts of a mesh surface, by modifying surface elements of the adaptedtraining surface model, (b) determining subvolumes of a training image,wherein the training image shows a training object by a training dataproviding unit and wherein the training surface model has been adaptedto the training object, for the surface elements of the modifiedtraining surface models, wherein for a respective surface element asubvolume is determined, which overlaps the respective surface element,(c) determining actual distances for the surface elements of themodified training surface models, wherein for a respective surfaceelement a respective distance to the un-modified training surface model,which has been adapted to the training object in the training image, isdetermined, and (d) training the convolutional neural network based onthe determined subvolumes and the determined actual distances.